{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/DeepLabCut/DeepLabCut/blob/master/examples/COLAB/COLAB_YOURDATA_TrainNetwork_VideoAnalysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RK255E7YoEIt"
   },
   "source": [
    "# DeepLabCut for your standard (single animal) projects!\n",
    "\n",
    "Some useful links:\n",
    "\n",
    "- [DeepLabCut's GitHub: github.com/DeepLabCut/DeepLabCut](https://github.com/DeepLabCut/DeepLabCut)\n",
    "- [DeepLabCut's Documentation: User Guide for Single Animal projects](https://deeplabcut.github.io/DeepLabCut/docs/standardDeepLabCut_UserGuide.html)\n",
    "\n",
    "\n",
    "This notebook illustrates how to use the cloud to:\n",
    "- create a training set\n",
    "- train a network\n",
    "- evaluate a network\n",
    "- create simple quality check plots\n",
    "- analyze novel videos!\n",
    "\n",
    "### This notebook assumes you already have a project folder with labeled data! \n",
    "\n",
    "This notebook demonstrates the necessary steps to use DeepLabCut for your own project.\n",
    "\n",
    "This shows the most simple code to do so, but many of the functions have additional features, so please check out the overview & the protocol paper!\n",
    "\n",
    "Nath\\*, Mathis\\* et al.: Using DeepLabCut for markerless pose estimation during behavior across species. Nature Protocols, 2019.\n",
    "\n",
    "\n",
    "Paper: https://www.nature.com/articles/s41596-019-0176-0\n",
    "\n",
    "Pre-print: https://www.biorxiv.org/content/biorxiv/early/2018/11/24/476531.full.pdf\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "txoddlM8hLKm"
   },
   "source": [
    "## First, go to \"Runtime\" ->\"change runtime type\"->select \"Python3\", and then select \"GPU\"\n",
    "\n",
    "As the COLAB environments were updated to CUDA 12.X and Python 3.11, we need to install DeepLabCut and TensorFlow in a distinct way to get TensorFlow to connect to the GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this will take a couple of minutes to install all the dependencies!\n",
    "!pip install --pre deeplabcut"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "25wSj6TlVclR"
   },
   "source": [
    "**(Be sure to click \"RESTART RUNTIME\" if it is displayed above before moving on !)** You will see this button at the output of the cells above ^."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "oTwAcbq2-FZz",
    "outputId": "9cfd8dcf-a0a8-4801-ed1d-fbcd5ec056af"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DLC loaded in light mode; you cannot use any GUI (labeling, relabeling and standalone GUI)\n"
     ]
    }
   ],
   "source": [
    "import deeplabcut"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cQ-nlTkri4HZ"
   },
   "source": [
    "## Link your Google Drive (with your labeled data, or the demo data):\n",
    "\n",
    "### First, place your project folder into you google drive! \"i.e. move the folder named \"Project-YourName-TheDate\" into google drive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KS4Q4UkR9rgG"
   },
   "outputs": [],
   "source": [
    "# Now, let's link to your GoogleDrive. Run this cell and follow the authorization instructions:\n",
    "# (We recommend putting a copy of the github repo in your google drive if you are using the demo \"examples\")\n",
    "\n",
    "from google.colab import drive\n",
    "\n",
    "drive.mount(\"/content/drive\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Frnj1RVDyEqs"
   },
   "source": [
    "YOU WILL NEED TO EDIT THE PROJECT PATH **in the config.yaml file** TO BE SET TO YOUR GOOGLE DRIVE LINK!\n",
    "\n",
    "Typically, this will be: `/content/drive/My Drive/yourProjectFolderName`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vhENAlQnFENJ"
   },
   "outputs": [],
   "source": [
    "# PLEASE EDIT THIS:\n",
    "project_folder_name = \"MontBlanc-Daniel-2019-12-16\"\n",
    "video_type = \"mp4\" #, mp4, MOV, or avi, whatever you uploaded!\n",
    "\n",
    "# No need to edit this, we are going to assume you put videos you want to analyze\n",
    "# in the \"videos\" folder, but if this is NOT true, edit below:\n",
    "videofile_path = [f\"/content/drive/My Drive/{project_folder_name}/videos/\"]\n",
    "print(videofile_path)\n",
    "\n",
    "# The prediction files and labeled videos will be saved in this `labeled-videos` folder\n",
    "# in your project folder; if you want them elsewhere, you can edit this;\n",
    "# if you want the output files in the same folder as the videos, set this to an empty string.\n",
    "destfolder = f\"/content/drive/My Drive/{project_folder_name}/labeled-videos\"\n",
    "\n",
    "#No need to edit this, as you set it when you passed the ProjectFolderName (above):\n",
    "path_config_file = f\"/content/drive/My Drive/{project_folder_name}/config.yaml\"\n",
    "print(path_config_file)\n",
    "\n",
    "# This creates a path variable that links to your Google Drive project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xNi9s1dboEJN"
   },
   "source": [
    "## Create a training dataset:\n",
    "\n",
    "### You must do this step inside of Colab\n",
    "\n",
    "After running this script the training dataset is created and saved in the project directory under the subdirectory **'training-datasets'**\n",
    "\n",
    "This function also creates new subdirectories under **dlc-models-pytorch** and appends the project config.yaml file with the correct path to the training and testing pose configuration file. These files hold the parameters for training the network. Such an example file is provided with the toolbox and named as **pytorch_config.yaml**.\n",
    "\n",
    "Now it is the time to start training the network!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "eMeUwgxPoEJP",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# There are many more functions you can set here, including which network to use!\n",
    "# Check the docstring for `create_training_dataset` for all options you can use!\n",
    "\n",
    "deeplabcut.create_training_dataset(\n",
    "  path_config_file, net_type=\"resnet_50\", engine=deeplabcut.Engine.PYTORCH\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "c4FczXGDoEJU"
   },
   "source": [
    "## Start training:\n",
    "This function trains the network for a specific shuffle of the training dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_pOvDq_2oEJW"
   },
   "outputs": [],
   "source": [
    "# Let's also change the display and save_epochs just in case Colab takes away\n",
    "# the GPU... If that happens, you can reload from a saved point using the\n",
    "# `snapshot_path` argument to `deeplabcut.train_network`:\n",
    "#   deeplabcut.train_network(..., snapshot_path=\"/content/.../snapshot-050.pt\")\n",
    "\n",
    "# Typically, you want to train to ~200 epochs. We set the batch size to 8 to\n",
    "# utilize the GPU's capabilities.\n",
    "\n",
    "# More info and there are more things you can set:\n",
    "#   https://deeplabcut.github.io/DeepLabCut/docs/standardDeepLabCut_UserGuide.html#g-train-the-network\n",
    "\n",
    "deeplabcut.train_network(\n",
    "    path_config_file,\n",
    "    shuffle=1,\n",
    "    save_epochs=5,\n",
    "    epochs=200,\n",
    "    batch_size=8,\n",
    ")\n",
    "\n",
    "# This will run until you stop it (CTRL+C), or hit \"STOP\" icon, or when it hits the end."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RiDwIVf5-3H_"
   },
   "source": [
    "Note, that **when you hit \"STOP\" you will get a `KeyboardInterrupt` \"error\"! No worries! :)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xZygsb2DoEJc"
   },
   "source": [
    "## Start evaluating:\n",
    "This function evaluates a trained model for a specific shuffle/shuffles at a particular state or all the states on the data set (images)\n",
    "and stores the results as .csv file in a subdirectory under **evaluation-results-pytorch**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nv4zlbrnoEJg"
   },
   "outputs": [],
   "source": [
    "deeplabcut.evaluate_network(path_config_file, plotting=True)\n",
    "\n",
    "# Here you want to see a low pixel error! Of course, it can only be as\n",
    "# good as the labeler, so be sure your labels are good!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BaLBl3TQtrfB"
   },
   "source": [
    "## There is an optional refinement step you can do outside of Colab:\n",
    "- if your pixel errors are not low enough, please check out the protocol guide on how to refine your network!\n",
    "- You will need to adjust the labels **outside of Colab!** We recommend coming back to train and analyze videos... \n",
    "- Please see the repo and protocol instructions on how to refine your data!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OVFLSKKfoEJk"
   },
   "source": [
    "## Start Analyzing videos: \n",
    "This function analyzes the new video. The user can choose the best model from the evaluation results and specify the correct snapshot index for the variable **snapshotindex** in the **config.yaml** file. Otherwise, by default the most recent snapshot is used to analyse the video.\n",
    "\n",
    "The results are stored in hd5 file in the same directory where the video resides. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Y_LZiS_0oEJl"
   },
   "outputs": [],
   "source": [
    "deeplabcut.analyze_videos(\n",
    "    path_config_file,\n",
    "    videofile_path,\n",
    "    videotype=video_type,\n",
    "    destfolder=destfolder,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8GTiuJESoEKH"
   },
   "source": [
    "## Plot the trajectories of the analyzed videos:\n",
    "This function plots the trajectories of all the body parts across the entire video. Each body part is identified by a unique color."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gX21zZbXoEKJ"
   },
   "outputs": [],
   "source": [
    "deeplabcut.plot_trajectories(\n",
    "    path_config_file,\n",
    "    videofile_path,\n",
    "    videotype=video_type,\n",
    "    destfolder=destfolder,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pqaCw15v8EmB"
   },
   "source": [
    "Now you can look at the plot-poses file and check the \"plot-likelihood.png\" might want to change the \"p-cutoff\" in the config.yaml file so that you have only high confidnece points plotted in the video. i.e. ~0.8 or 0.9. The current default is 0.4. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pCrUvQIvoEKD"
   },
   "source": [
    "## Create labeled video:\n",
    "This function is for visualization purpose and can be used to create a video in .mp4 format with labels predicted by the network. This video is saved in the same directory where the original video resides. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6aDF7Q7KoEKE"
   },
   "outputs": [],
   "source": [
    "deeplabcut.create_labeled_video(\n",
    "    path_config_file,\n",
    "    videofile_path,\n",
    "    videotype=video_type,\n",
    "    destfolder=destfolder,\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "Copy of latest_Colab_TrainNetwork_VideoAnalysis.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3.8.12 ('dlc')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.8.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "70cad038f2bddb56e8a0ba66c48b76ebce20579892bf83e71733a81977e3ceea"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
